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Abstract.  Several  techniques  for  evaluating  a  groupware  interface  have  emerged  in  recent  years. 
Though  a  groupware  is  considered  to  be  difficult  to  evaluate,  many  researchers  to  date  have  attempted 
to  uncover  the  usability  problems  of  the  interface  using  multiple  evaluation  methods  or  triangulation 
running  parallel  to  each  other.  The  aims  of  this  paper  are  to  compare  the  effectiveness  of  dual 
evaluation  methods  such  as  Usability  Testing  and  Heuristic  Evaluation  on  different  groupware 
products  in  the  organizations  of  Pakistan  and  to  manifest  the  user’s  preference  for  an  ideal  interface. 
Because  of  the  paucity  of  resources,  the  user  based  and  inspection  based  techniques  couldn’t  be  carried 
out  in  other  parts  of  Pakistan  where  the  IT  industry  has  mushroomed  in  the  past  few  years. 
Nevertheless  these  restrictions  didn’t  significantly  affect  from  validity  of  the  findings.  Upon 
comparing  the  results  from  the  two  Usability  Evaluation  Methods  (UEM),  we  concluded  that  both 
UEMs  were  equally  efficient  in  identifying  potential  usability  problems. 

Keywords.  Usability  evaluation  methods,  Heuristic  evaluation.  Usability  testing.  Groupware. 


1.  Introduction 

There  are  a  variety  of  Knowledge  Management  (KM)  tools 
which  make  communication  and  collaboration  among 
geographically  dispersed  staff  possible.  The  collaboration 
tools  help  people  work  together  and  share  the  information  that 
has  been  managed  by  Knowledge  Management.  KM  systems 
tend  to  fail  in  the  absence  of  effective  user  interface  design, 
regardless  of  the  value  of  knowledge  stored  in  it.  A  good  user 
interface  is  an  improved  technology  that  allows  users  to 
effectively  perceive  and  express  information.  The  interface 
design  plays  a  great  role  in  increasing  the  team  productivity  of 
work  and  presents  information  to  users  with  more  accuracy 
and  a  higher  level  of  control. 

Software  tools  that  aid  in  communications  and 
collaboration  are  included  within  knowledge  management 
because  of  their  role  in  facilitating  the  flow  of  tacit 
information  (Nonakia  and  Takeuchi,  1995).  Groupware  is 
simply  a  collaboration  tool  that  helps  people  work  together 
more  easily  or  more  effectively.  It  typically  allows  them  to 
communicate,  coordinate  and  collaborate  (Hills,  1997). 
Groupware  interface  has  been  an  area  of  great  concern  for 
many  researchers  as  it  is  considered  difficult  to  evaluate 
because  of  its  multiple-user  nature.  Numerous  researches  have 
been  done  using  single  evaluation  and  multiple  evaluation 
methods  including  both  inspection  based  techniques  and  user 
based  techniques.  In  spite  of  that,  there  are  no  set  standards  for 
using  any  particular  technique.  Evaluating  a  groupware 
interface  is  an  evolving  process  and  requires  more 
investigation  on  the  suitability  of  techniques  applied  in 
different  culture  contexts. 

Usability  Evaluation  (UE)  consists  of  methodologies 
for  measuring  the  usability  aspects  of  a  system’s  user  interface 


(UI)  and  identifies  specific  problems  (Dix  et  ah,  1998  and 
Nielson,  1993).  A  Usability  Problem  (UP)  is  defined  as  a  flaw 
in  the  designing  of  a  system  that  makes  the  attainment  of  a 
particular  goal  with  the  use  of  the  system  ineffective  and/or 
inefficient,  and  thus  lowers  the  user’s  level  of  satisfaction  with 
its  usage.  Until  now,  a  number  of  researches  have  been  carried 
out  comparing  the  effectiveness  of  different  UEMs.  (Bailey  et 
ah,  1992;  Jeffries  et  ah,  1991;  Karat  et  ah,  1992;  Desurvire, 
1994;  Jorgensen,  1999). 

Complementary  methods,  converging  measures,  and 
triangulation  are  embraced  by  most  social  science 
methodologists  (Law  and  Hvannberg,  2002;  Eisner  and 
peshkin,  1990;  Sullivan,  1991;  Yin,  1984).  Based  on  the 
research  studies  done  earlier  it  has  been  found  that  Heuristic 
evaluation  (HE)  and  usability  testing  (UT)  complemented  each 
other  (Gutwin  et  ah,  2001)  and  proved  to  be  a  good 
combination  to  uncover  minor  and  major  software  bugs.  The 
reason  for  adopting  multiple  methods  is  to  assemble  evidence 
from  more  than  one  source  of  information.  The  combined  use 
of  these  two  methods  would  reveal  many  hidden  problems, 
which  might  be  overlooked  when  one  technique  is  used. 

In  the  remainder  of  this  paper,  the  main  results  of  the 
evaluations  are  drawn  and  compared.  We  also  highlight  the 
highest  rated  and  least  rated  usability  problems  within  each 
organization. 

2.  Objectives  and  Scope 

This  study  sought  to  identify  usability  problems  of  the  KM 
groupware  used  by  the  selected  leading  organizations  in 
Pakistan.  It  also  tried  to  determine  the  usability  problems  of 
groupware  interface  by  conducting  the  heuristic  evaluation 
technique,  that  is,  to  judge  its  compliance  with  Jakob 
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Nielsen’s  ten  heuristics.  Similarly  it  determined  the  usability 
problems  of  groupware  interface  by  applying  the  usability 
testing  method. 

At  the  end,  this  study  attempted  to  compare  the  results 
of  the  two  methods  and  derived  some  conclusions  on  the 
effectiveness  of  each  technique:  user-based  technique  vs.  the 
inspection  method. 

Standard  procedures  based  on  UT  and  HE  literatures 
were  adopted  when  conducting  both  techniques.  The  research 
was  conducted  in  companies  belonging  to  different  corporate 
sectors  in  Pakistan.  The  companies  covered  in  Pakistan  were 
Askari  Commercial  Bank  (ACB),  LMK  Resources  (LMKR) 
and  Enterprise  Technology  Pvt.  Ltd  (ETPL).  The  only 
consideration  for  selecting  the  organization  was  the 
availability  of  groupware  product.  Table  1  shows  the 
companies  selected  and  the  groupware  product  in  use. 


Table  1.  Company  profiles 


Company 

Groupware  Product 

LMK  Resources 

Lotus  Notes 

Askari  Commercial  Bank 

Askari  In-house  product 

Enterprise  Technologies  Pvt. 
Ltd. 

Simex  (In-house  product) 

The  recommended  number  of  evaluators  for  a  HE  is 
between  three  and  five,  given  that  the  informational  gain  with 
an  additional  evaluator  drastically  decreases  after  the  fifth  one 
and  that  the  benefit-cost  ratio  is  highest  when  three  or  four 
evaluators  are  employed  (Nielsen,  1994).  But  in  this  study  the 
number  of  Evaluators  for  HE  was  two  instead  of  the 
traditional  three  to  five.  This  is  supported  again  by  Nielsen’s 
(1995)  theory  that  found  that  the  highest  percent  of  the 
respondents  employed  two  evaluators  for  HE,  although  they 
were  instructed  to  use  three  to  five.  Similarly  based  on 
Nielsen’s  theory,  Law  and  Hvannberg  (2002)  involved  only 
two  evaluators  for  HE  in  their  multiple  method  study. 

3.  Method  and  Results 

3.1.  Heuristic  evaluation  assessment 

HE  was  performed  on  the  user  interface  design  of  groupware 
by  two  evaluators  (El  and  E2)  taken  from  each  organization. 
Both  inspectors  performed  the  evaluation  independently  and 
afterwards  an  average  of  the  usability  problems  identified  by 
them  was  obtained.  The  inspectors  were  given  considerable 
freedom  on  how  and  when  they  perform  their  task,  quite 
similar  to  Nielson’s  practice  of  conducting  HE.  Since  the 
evaluators  were  taken  from  the  organization  itself,  there  was 
no  issue  of  non-familiarity  with  the  groupware  product.  The 
inspection  method  identified  several  potential  UP  in  the 
groupware  interface.  The  three  most  obvious  or  highly  rated 
UP  found  within  each  organization  have  been  highlighted  and 
discussed.  Table  2  depicts  the  profile  of  the  selected 
evaluators  for  HE  within  each  organization. 


Table  2.  Profile  of  the  Evaluators  for  HE 


Co.* 

Eva* 

Designation 

Age 

Exp.* 

ETPL 

El 

Senior 

Programmer 

26 

10  years 

E2 

Software  Engineer 

28 

5  years 

LMKR 

El 

Testing  Analyst 

26 

1 1  months 

E2 

Testing  Analyst 

26 

2  years 

ACB 

El 

Junior  officer 

21 

2  years 

E2 

Junior  officer 

28 

1  year 

Note:  *Co=  Company,  *Eva=  Evaluator  &  *Exp=  Experience  of  working 


Table  3  shows  the  percentage  of  usability  problems 
(UP)  identified  by  each  evaluator  in  Enterprise  Technologies 
Pvt.  Ltd  and  the  average  percentage  is  shown  in  the  third 
column.  In  ETPL,  where  an  in-house  developed  groupware 
SIMEX  (secure  internet  message  exchange)  is  being  used,  the 
evaluation  of  the  interface  exposed  more  than  50%  of  the 
usability  problems  in  the  ‘error  prevention’,  ‘help  and 
documentation’  and  ‘recognition  rather  than  recall’  areas.  This 
implies  that  one  of  the  main  problems  with  the  interface  is  that 
it  does  not  provide  features  that  can  prevent  users  from 
committing  errors  during  the  interaction.  Help  and 
documentation  was  not  provided  in  details  for  the  ease  of  use 
of  the  system.  Another  usability  problem  issue  is  pertaining  to 
the  instructions  for  the  use  of  the  systems,  which  were  neither 
visible  nor  retrievable  to  the  users. 


Table  3.  Results  of  Heuristic  Evaluation  at  Enterprise 
Technologies  Pvt.  Ltd. 


HEURISTICS 

E  1 

E  2 

Ave* 

Error  Prevention 

80% 

50% 

65% 

Help  and  Documentation 

57% 

50% 

54% 

Recognition  Rather  Than  Recall 

50% 

55% 

53% 

Visibility  of  System  Status 

43% 

50% 

47% 

Aesthetic  and  Minimalist  Design 

44% 

40% 

42% 

Match  Between  System  and  the  Real  World 

40% 

30% 

35% 

User  Control  and  Freedom 

33% 

33% 

33% 

Help  Users  Recognize,  Diagnose,  and 

Recover  From  Errors 

33% 

30% 

32% 

Consistency  and  Standards 

36% 

25% 

31% 

Flexibility  and  Minimalist  Design 

29% 

30% 

30% 

Note:  *Ave=  Average 


The  HE  of  Lotus  Notes  at  LMK  Resources  shows  that 
more  than  50%  of  the  usability  problems  found  were  related  to 
‘match  between  the  system  and  the  real  world’,  ‘recognition 
rather  than  recall’  and  ‘visibility  of  the  system’s  status’. 

Usability  problems  found  related  to  having  a  less 
logical  order  of  the  information  and  system  oriented  terms 
rather  than  user-oriented  terms.  Another  problem  was  with 
instructions  for  use,  which  were  less  visible  and  retrievable. 
Yet  another  usability  problem  which  cannot  be  ignored,  shows 
that  the  system  doesn’t  keep  the  users  informed  about  the 
action  taken  and  doesn’t  give  appropriate  feedback  with  in  the 
given  time. 
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Table  4.  Results  of  Heuristic  Evaluation  at  LMK  Resources 


HEURISTICS 

E  1 

E  2 

Ave* 

Match  Between  System  and  the  Real 
World 

75% 

57% 

66% 

Recognition  Rather  Than  Recall 

69% 

50% 

60% 

Visibility  of  System  Status 

58% 

47% 

53% 

Error  Prevention 

75% 

25% 

50% 

Help  Users  Recognize,  Diagnose,  and 
Recover  From  Errors 

64% 

36% 

50% 

Aesthetic  and  Minimalist  Design 

67% 

22% 

45% 

Consistency  and  Standards 

55% 

32% 

44% 

Help  and  Documentation 

75% 

5% 

40% 

Flexibility  and  Minimalist  Design 

0% 

57% 

29% 

User  Control  and  Freedom 

0% 

25% 

13% 

Note:  *Ave=Average 


Similarly,  another  pair  of  evaluators  from  the  Askari 
Commercial  Bank  inspected  their  bank’s  in-house  developed 
groupware  and  the  results  derived  from  their  evaluation  are 
summarized  in  Table  5. 

As  shown  in  table  a  majority  of  the  usability  problems 
(more  than  50%)  lie  with  the  ‘Help  users  recognize,  Diagnose, 
and  Recover  From  Errors’  and  ‘visibility  of  the  system’s 
status’.  The  interface  of  the  groupware  displayed  the  error 
messages  using  complicated  terms  rather  than  plain  language. 
This  aspect  of  interface  problem  stated  that  recovery  from 
error  and  appropriate  error  messages  were  not  provided  in 
most  of  the  cases.  It  neither  indicates  the  problem  precisely 
nor  does  it  give  a  constructive  solution.  Another  weak  aspect 
of  the  interface  found  by  the  evaluators  was  that  the  system 
doesn’t  keep  the  user  informed  about  the  processes  while 
running  their  tasks  and  appropriate  feedbacks  of  their 
operations. 


50%.  Each  product  had  a  few  high  percentages  of  interface 
related  problems  pertaining  to  certain  aspects. 


Table  6.  Comparison  of  the  results  of  HE  among  Enterprise 
Technologies  Pvt.  Ltd.  (ETPL),  LMK  Resources  (LMKR)  and 
Askari  Commercial  Bank  (ACB) 


HEURISTICS 

ETPL 

LMKR 

ACB 

Visibility  of  the  System  Status 

47% 

53% 

50% 

Match  Between  System  and  the 
Real  World 

35% 

66% 

15% 

User  Control  and  Freedom 

33% 

13% 

44% 

Consistency  and  Standards 

31% 

44% 

26% 

Error  Prevention 

65% 

50% 

44% 

Recognition  Rather  than  Recall 

53% 

60% 

41% 

Flexibility  and  Minimalist  Design 

30% 

29% 

42% 

Aesthetic  and  Minimalist  Design 

42% 

45% 

47% 

Help  User  recognize,  diagnose, 
and  recover  from  errors 

32% 

50% 

52% 

Help  and  documentation 

54% 

40% 

35% 

AVERAGE 

42  % 

45  % 

40  % 

3.2.  Usability’  testing  assessment 

The  previous  section  shows  the  results  of  HE.  This  section 
presents  findings  on  usability  problems  using  the  UT  method. 
We  limited  the  number  of  people  who  could  participate  in  the 
usability  testing  to  five  participants  from  each  organization. 
Their  evaluation  of  the  groupware’s  interface  was  based  on  the 
guidelines  for  User  Interface  Design  developed  by  Hix  and 
Hartson  (1993).  The  participants  in  the  UT  were  different 
from  those  in  the  inspection-based  technique  and  were 
unaware  of  the  results  of  that  evaluation.  Table  7  shows  the 
profile  of  the  UT  participants  at  each  organization. 


Table  5.  Results  of  Heuristic  Evaluation  Table  7-  Proflle  of  the  Participants  (P  1-P  5)  for  UT 

at  Askari  Commercial  Bank 


Table  6  highlights  the  usability  problems  identified 
using  the  HE  method  at  each  organization.  The  last  row  in  the 
table  shows  the  average  percentage  of  all  usability  problems 
obtained  from  each  organization.  LMK  Resources  having 
Lotus  Notes  as  their  groupware  product  showed  a  higher 
percentage  of  usability  problems  than  Enterprise’s  SIMEX  and 
Askari’s  in-house  product.  But  none  of  the  total  usability 
problems  found  in  the  interface  of  the  three  products  exceeded 


HEURISTICS 

E  1 

E  2 

Ave* 

Help  Users  Recognize,  Diagnose,  and 
Recover  From  Errors 

63% 

40% 

52% 

Visibility  of  System  Status 

33% 

67% 

50% 

Aesthetic  and  Minimalist  Design 

33% 

60% 

47% 

User  Control  and  Freedom 

20% 

67% 

44% 

Error  Prevention 

38% 

50% 

44% 

Flexibility  and  Minimalist  Design 

17% 

67% 

42% 

Recognition  Rather  Than  Recall 

24% 

58% 

41% 

Help  and  Documentation 

19% 

50% 

35% 

Consistency  and  Standards 

26% 

25% 

26% 

Match  Between  System  and  the  Real 
World 

29% 

0% 

15% 

Note:  *Ave=  Average 


Co.* 

Designation 

Age 

Exp* 

(Years) 

ETPL 

PI 

Programmer 

24 

4 

P2 

Senior  Programmer 

25 

5 

P3 

Programmer 

25 

2 

P4 

Senior  Programmer 

27 

4 

P5 

Software 

Programmer 

26 

3 

LMKR 

PI 

Software  Engineer 

27 

4 

P2 

Network 

Administrator 

27 

5 

P3 

Testing  Analyst 

26 

2 

P4 

Software 

Programmer 

24 

3 

P5 

Software 

Programmer 

24 

2 

ACB 

PI 

Junior  officer 

21 

2 

P2 

Junior  officer 

28 

1 

P3 

Senior  officer 

31 

5 

P4 

Programmer 

25 

2 

P5 

Programmer 

27 

4 

Note:  *Co=  Company  &  *Exp=Experience  of  working 
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Table  8  shows  the  evaluation  of  interface  by  five 
participants  at  Enterprise  Technologies  Pvt.  Ltd.  The  average 
result  is  shown  in  the  last  column.  The  highest  percentages  of 
usability  problems  found  were  related  to  ‘individual 
differences’,  ‘cognitive  directness’,  and  ‘attention’.  The 
Usability  problem  found  indicates  that  the  system’s  interface 
had  lesser  customization  facility  of  screen  layouts,  which 
doesn’t  accommodate  individual  difference  in  user  experience. 
Though  it  is  not  preferable  to  use  attention-grabbing 
techniques  but  these  were  over  used  to  some  extent.  The  users 
also  underscored  in  their  evaluation  the  over  usage  of  less 
meaningful  icons  and  letters. 


Table  8.  Results  for  Usability  Testing  at  Enteiprise 
Technologies  Pvt  Ltd. 


Ave 

Usability  Guidelines 

p  i 

P  2 

P  3 

P  4 

P  5 

* 

100 

100 

100 

100 

100 

100 

Individual  Differences 

% 

% 

% 

% 

% 

% 

100 

0% 

100 

100 

100 

80% 

Cognitive  Directness 

% 

% 

% 

% 

Attention 

67% 

100 

67% 

67% 

33% 

67% 

% 

Color  Scheme 

50% 

40% 

60% 

40% 

40% 

46% 

Feedback 

33% 

67% 

33% 

33% 

33% 

40% 

Display  Issues 

40% 

0% 

50% 

40% 

40% 

34% 

Human  Memory 
Limitation 

50% 

0% 

50% 

25% 

0% 

25% 

Simplicity 

33% 

0% 

33% 

33% 

0% 

20% 

System  Messages 

0% 

50% 

0% 

0% 

25% 

15% 

Consistency 

0% 

20% 

0% 

0% 

0% 

4% 

Modality 

0% 

0% 

0% 

0% 

0% 

0% 

Anthropomorphization 

0% 

0% 

0% 

0% 

0% 

0% 

Note:  *Ave=  Average 


In  LMK  Resources  the  results  showed  a  majority  of  the 
problems  related  to  ‘individual  differences’  and  ‘modality’.  As 
shown  in  Table  9  the  results  indicate  that  the  degree  of 
customization  is  not  provided  in  the  dialogue  elements.  The 
other  usability  problem  refers  to  the  lack  of  having  irreversible 
commands  and  escape  routes  for  the  user  if  he  wishes  to  opt 
out  of  the  system  during  the  process. 


Table  9.  Results  for  Usability  Testing  at  LMK  Resources 


Usability  Guidelines 

p  i 

P  2 

P  3 

P  4 

P  5 

Ave* 

Individual  Differences 

33% 

50% 

75% 

33% 

100 

% 

58% 

Modality 

100 

% 

67% 

100 

% 

0% 

0% 

53% 

Feedback 

0% 

67% 

67% 

33% 

0% 

33% 

Human  Memory 

Limitation 

0% 

25% 

25% 

100 

% 

0% 

30% 

System  Messages 

33% 

25% 

50% 

33% 

0% 

28% 

Simplicity 

33% 

0% 

33% 

33% 

33% 

26% 

Cognitive  Directness 

100 

% 

0% 

0% 

0% 

0% 

20% 

Color  Scheme 

40% 

20% 

0% 

20% 

0% 

16% 

Attention 

67% 

0% 

0% 

0% 

0% 

13% 

Consistency 

60% 

0% 

0% 

0% 

0% 

12% 

Display  Issues 

40% 

0% 

20% 

0% 

0% 

12% 

Anthropomorphization 

0% 

0% 

0% 

0% 

0% 

0% 

Note:  *Ave= Average 


The  groupware  in  Askari  Commercial  Bank  has  the 
most  usability  problems  of  all,  by  a  having  majority  of  the 
problems  with  ‘individual  differences’,  ‘cognitive  directness, 
‘system  messages’,  ‘feedback’  and  ‘display  issue’.  As  shown 
in  the  Table  10  ‘individual  difference’  was  the  most 
problematic  issue  of  the  interface.  A  majority  of  the  user 
participants  preferred  to  have  more  user  preference  for  screen 
layout  and  appearance  of  icons.  The  second  highest  rated 
usability  problem  was  the  ‘cognitive  directness’,  which 
demands  the  system  to  use  meaningful  icons/  letters 
throughout  the  system.  Lastly  the  problem  that  concerned  the 
users  was  the  ambiguous  messages  that  were  displayed  on 
several  occasions.  They  often  encountered  alarming  messages 
that  lacked  specific  and  constructive  words.  The  system  also 
lacked  appropriate  status  indicators  to  show  the  user  the 
progress  with  a  lengthy  operation  and  appropriate  articulatory 
feedback.  The  last  issue  concerned  the  users  regarding  the 
interface  was  the  display  issue,  which  refers  to  maintaining 
display  inertia.  It  regards  to  the  changes  from  one  screen  to  the 
next  within  a  functional  task  situation.  The  users  found  the 
screen  layout  being  less  balanced  with  information  not 
grouped  logically. 
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Table  10.  Results  for  Usability  Testing  at  Askari  Commercial 

Bank 


Usability  Guidelines 

p  i 

P  2 

P  3 

P  4 

P  5 

Ave 

* 

Individual  Differences 

100 

% 

100 

% 

75% 

100 

% 

100% 

95% 

Cognitive  Directness 

100 

% 

0% 

100 

% 

100 

% 

100% 

80% 

System  Messages 

100 

% 

0% 

50% 

100 

% 

100% 

70% 

Feedback 

too 

% 

50% 

50% 

67% 

67% 

66% 

Display  Issues 

40% 

75% 

40% 

40% 

60% 

51% 

Color  Scheme 

60% 

20% 

40% 

40% 

60% 

44% 

Human  Memory 
Limitation 

75% 

25% 

25% 

25% 

50% 

40% 

Modality 

33% 

50% 

33% 

33% 

33% 

36% 

Consistency 

20% 

33% 

20% 

20% 

33% 

25% 

Simplicity 

0% 

0% 

0% 

33% 

33% 

13% 

Attention 

0% 

0% 

0% 

0% 

33% 

7% 

Anthropomorphization 

0% 

0% 

0% 

0% 

0% 

0% 

Note:  *Ave=Average 


Table  1 1  shows  the  highlighted  percentage  of  usability 
problems  of  interface  found  within  each  participating 
organization.  The  interface  of  Askari  Bank’s  in-house 
groupware  shows  the  highest  percentage  of  usability 
problems,  whereas  LMK  Resources  shows  the  least 
percentage  of  usability  problems.  Among  the  highly  rated 
usability  problems,  ‘individual  difference’  was  found  to  be  the 
highest  problematic  element  of  the  interface  in  all  the  three- 
groupware  products.  The  users  were  not  satisfied  with  the 
degree  of  customization  of  screen  layout  and  icons  according 
to  their  individual  preference.  ‘Anthromorphization’  was  least 
rated  among  other  usability  problems  discovered  in  the 
interface  of  all  the  three  groupware  products.  This  aspect 
relates  to  the  attribution  of  human  characteristics  to  objects. 
Consistency  of  the  icons,  terminologies  and  color  scheme  was 
another  issue,  which  was  not  evaluated  by  the  users  as  a  major 
usability  problem.  ‘Consistency’  had  been  maintained  in  the 
interface  of  SIMEX  and  Lotus  Notes. 


Table  11.  Comparison  of  the  results  of  UT  among  Enterprise 
Technologies  Pvt.  Ltd.,  LMK  Resources  and  Askari 
Commercial  Bank 


UT  GUIDELINES 

ETPL 

LMKR 

ACB 

Consistency 

4% 

12% 

25% 

Simplicity 

20% 

26% 

13% 

Human  Memory 
Limitation 

25% 

30% 

40% 

Cognitive  Directness 

80% 

20% 

80% 

Feedback 

40% 

33% 

67% 

System  message 

15% 

28% 

70% 

Anthromorphization 

0% 

0% 

0% 

Modality 

0% 

53% 

36% 

Attention 

67% 

13% 

7% 

Display  issue 

34% 

12% 

51% 

Individual  differences 

100% 

58% 

95% 

Color  scheme 

46% 

16% 

44% 

AVERAGE 

36% 

25% 

44% 

4.  Comparisons  and  Discussion 

After  obtaining  the  results  of  HE  and  UT,  a  comparison  of  the 
usability  problems  was  made  within  each  organization.  Table 
12  shows  the  total  usability  problems  found  through  using  HE 
and  UT.  The  last  column  shows  the  average  of  HE  and  UT 
found  within  each  participating  organization. 


Table  12.  Comparison  of  both  techniques 


COMPANIES 

UEM 

ETPL 

LMKR 

ACB 

Average 

HE 

42% 

45% 

40% 

42% 

UT 

36% 

25% 

44% 

35% 

The  average  percentages  of  usability  problems  (UP) 
identified  by  HE  and  UT  are  42%  and  35%  respectively. 
Combining  the  results  obtained  in  all  the  three  organizations, 
HE  found  a  higher  percentage  of  usability  problems, 
exceeding  UT  by  7%.  Overall  it  can  be  inferred  that  HE  as 
applied  in  this  study  has  produced  the  best  results  as  it 
exposed  more  usability  problems  than  UT  and  both  yielded 
different  types  of  usability  problems.  Our  intent  for  using  HE 
was  to  carry  out  a  broad  inspection  that  would  look  into  every 
aspect  of  groupware  interface. 

Comparing  both  Usability  Evaluation  Methods  within 
individual  company  case  shows  somewhat  different  picture  for 
Askari  Commercial  bank.  The  percentage  of  usability 
problems  identified  by  UT  is  higher  than  HE.  The  evaluators 
might  have  overlooked  what  might  seem  to  be  a  potential 
problem  from  a  user’s  point  of  view  or  they  overestimated  the 
user’s  capabilities.  In  ETPL  and  LMKR  where  the  HE  yields 
higher  percentage  of  usability  problems  than  UT,  this  situation 
can  be  explained  from  two  perspectives,  evaluators’  and 
users’.  According  to  the  users  that  particular  aspect  of 
interface  may  not  have  looked  as  a  potential  weakness  to 
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them,  therefore  they  didn’t  consider  it  as  a  potential  threat.  As 
interpreted  by  Law  &  Hvannberg  (2002),  it  could  be  that  the 
UT  participants  were  unable  to  locate  the  usability  problems  in 
the  first  place.  Another  assessment  from  evaluators’  point  of 
view  is  that  HE  produced  false  positives,  which  are  minor 
bugs  that  do  not  negatively  impact  user  performance  or  the 
user’s  perception  of  product  quality. 

A  combination  of  inspection  based  and  user  based 
methods  was  found  to  be  advantageous  in  detecting  the 
interface  bugs.  Gutwin  et  al.  (2001)  specifically  supported  the 
two  techniques  (UT  and  HE)  working  in  combination.  Jeffries 
et  al.  (1991)  addressed  the  effectiveness  of  four  techniques 
namely:  HE,  UT,  Guidelines  and  Cognitive  Walkthrough  for 
evaluating  UI  and  inferred  that  HE  and  UT  produced  more 
serious  problems  than  the  other  techniques.  This  combination 
of  methods  was  also  adopted  in  the  study  carried  out  by  Law 
and  Hvannberg  (2002)  in  the  European  context,  focusing  on 
the  two  cultures,  Swiss  and  Icelandic.  This  study  is  unique  in 
its  own  way  that  no  detailed  research  has  been  done  in  Asian 
cultures  specifically  in  Pakistan. 

5.  Conclusion 

The  dual  evaluation  found  out  that  both  UEMs  have  a  valid 
role  in  groupware  evaluation.  The  two  types  of  techniques 
worked  well  in  combination  uncovering  several  potential 
problems,  which  couldn’t  be  revealed  by  using  only  one 
evaluation  method.  In  this  study  the  need  for  adopting  the  dual 
evaluation  method  is  not  only  to  measure  the  effectiveness  of 
each  technique  but  also  to  uncover  as  many  of  the  usability 
problems  as  possible.  The  evaluations  using  both  techniques 
showed  that  not  only  major  problems  have  been  uncovered  but 
the  minor  ones  came  to  light  as  well.  Even  though  HE  showed 
false  positive  in  two  cases,  overall  it  helped  in  addressing  bugs 
that  impact  user  performance  and  satisfaction.  To  have  more 
effective  detection  of  usability  problem,  the  UI  evaluators  and 
practitioners  can  use  the  strengths  of  these  two  methods 
carefully. 

Though  the  study  focuses  on  Pakistani  organizations, 
and  the  results  specifically  reflect  the  situations  of  these 
organizations,  the  findings  may  very  well  be  applicable  to 
organizations  in  other  developing  countries  in  Asia. 
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